feat(routing): automatic prompt caching + a per-session routing budget#982
feat(routing): automatic prompt caching + a per-session routing budget#982Spherrrical wants to merge 15 commits into
Conversation
Reconcile prompt caching with intelligent routing: zero-config implicit session affinity, cache_control preservation/injection across transforms, cache-adjusted routing economics, and a response-driven cache-hit feedback loop.
# Conflicts: # crates/brightstaff/src/handlers/llm/mod.rs # crates/brightstaff/src/router/model_metrics.rs # crates/common/src/configuration.rs
Spherrrical
changed the title
Add opt-in session_stickiness.record_counterfactual that emits the plano.switch.counterfactual_route OTEL attribute when the cache-regret gate retains the previous model, capturing the route it would have taken had the switch been allowed. Telemetry only; the candidate is never dispatched. Includes schema, demo config, and a getting-started guide.
|
@Spherrrical Thanks for pushing this PR!
|
…sive modules Extract the two concerns interleaved in the LLM handler into dedicated sibling modules for readability: prompt_caching.rs (cache-marker injection) and session_stickiness.rs (session key/prefix-hash resolution, pin lookup, cache-regret gate, and pin planning). handlers/llm/mod.rs is now a thin orchestrator that calls them at clear phase boundaries. Behavior-preserving.
It's not limited to OpenAI + Anthropic. The strategy is resolved per (gateway × model family × upstream API) in cache_marker_strategy: OpenAI-family caches automatically (no markers) across OpenAI/Azure/etc.; Anthropic-family gets native Messages breakpoints or, behind OpenAI-compatible gateways (DigitalOcean, OpenRouter), content-part cache_control; unimplemented backends (Bedrock) are an honest None. Crucially, we don't assume caching worked, we read it back from each provider's own response usage
Done!
The only change there is a one-line because the caching work added two fields to the shared Usage struct so every existing literal had to initialize them. It's not a revival of Arch-Function. The handler is technically still wired (/function_calling route), so if it's truly deprecated we can remove it in a separate cleanup PR rather than bundling that here.
It changed because session stickiness introduced stale pins: get() now returns
Done! |
Why only the first user message in determining the prompt prefix? Separately, I don't follow this notion of "auto injected". That whole sentence is slop.
How would we know that the cache is "plausibly" still warm? And I don't follow this whole regret cost gating math one bit. Not clear to me, how will it be clear to the developers building with plano.
I am not sure what this means? how does this work? what does the developer have to do? Separately I am not sure how can you compute prefix trees unless you are storing the prompt prefix somewhere in a database Please share the user experience in the config.yaml here. Don't link to the demo.
|
Replace observed_cache_hit + per-request threshold with time/provider-TTL warmth and a per-session switch budget in a single session_router::route() used by both the full-proxy and /routing decision paths.
Spherrrical
changed the title
|
Spherrrical
changed the title
|
Those are actually two distinct effects, and both apply: Free = the switch is allowed regardless of budget. Any switch with cost <= 0 is permitted unconditionally and it never touches the remaining budget as a gate. This is the Refund (credit_negative, default true) = on top of being allowed, we grow the remaining budget by the magnitude of the savings (budget -= cost, in which cost is negative). The refund exists because the budget is meant to be a running ledger of the session's net switch cost.
Routing is an already existing top level object, we're just adding routing_budget to it now. We've set it to presence based for enable/disable. If routing_budget is absent, then the router has honored pick. If it's present, then the gate is active. If you have it set to 0, then that means it's never pay to switch. If it's set to "unlimited" (a string value), it makes more developer experience sense to just remove it to disable it.
Will report back on this. |
|
@Spherrrical We can't have this refund logic.
|
Replace the absolute seed_usd pool with max_overhead_pct (whole-number percent) measured against the session's running never-switch baseline: allow a paid switch only while cumulative switch spend stays within max_overhead_pct% of what staying on the anchor would have cost. Track baseline_usd and switch_spend_usd on the binding (monotonic, no refunds).
…head cap Rename the switch-decision reasons (within_budget/over_budget -> within_cap/ over_cap) and the session span attributes to match the percentage overhead-cap model: budget_remaining_in_usd -> overhead_pct + switch_spend_in_usd + baseline_in_usd, and switch threshold_in_usd -> overhead_ceiling_in_usd.
Price each turn from the catalog rates and surface it as llm.usage.{input,
output,total}_cost_usd on the (llm) span, then accumulate into a conversation-
level session_cost_usd on the binding and emit plano.session.total_cost_in_usd
on the routing span. Cache-creation tokens are priced at the plain input rate,
and the OpenAI vs Anthropic prompt-token convention (cached folded in vs
reported separately) is captured at parse time so the cost math is correct for
both. Rates are resolved request-side since the response path is synchronous.
There was a problem hiding this comment.
- Why
est_context_tokens? We should use actual context tokens not the estimated ones? - If I understand correctly,
anchor_modelhere refers to the default model, not the one the request was sent to? The formula should be:
switch_cost_in_usd = context_tokens × (candidate_uncached_input_rate − current_cached_input_rate) / 1M
current_cached_input_rate is calculated based on the model that the request was sent to, and never-switch baseline should be calculated based on the default model for the session.
There was a problem hiding this comment.
- rename
estimate_context_tokens->actual_context_tokens. This is not an estimation. - rename
max_overhead_pct->max_routing_overhead? - distinguish between
default_modelfor the session andanchor_modelfor the one that handles the latest previous request
knn-do
left a comment
knn-do
left a comment
There was a problem hiding this comment.
@Spherrrical
I love the detailed documentation, thanks for adding support for the counterfactual measurement based on my request as well.
I reviewed the updated code again & have some more requests:
- Can we structure the code to allow other cache-aware routing policies besides this one? The current policy is too sensitive to the initial / local prompt & may stay in a sub-optimal route for an extended period of time. We need to fundamentally try different policies & measure impact to make data driven decisions here.
- Can we keep track of historical routes in the current session (bounded)? This can not only allow for new kinds of policies it should also allow for better switching cost estimates (explained below)
- The cache switching cost estimate may be too poor of an approximation & perhaps worth fixing based on data ^. And I don't just mean the fact that the tokenizer varies wildly between models (eg OAI is much more efficient than OpenModels & ANT). If we switch from model A -> B -> A, the switching cost is not the entire context that should be considered.
Overall, I want to understand what is the best way to test a bunch of policies quickly rather than have you work on documentation testing & observability for each.
We now have a way to measure quality / cost tradeoffs for router systematically.
I also want @salmanap 's thoughts on whether we want to maintain multiple strategies in Plano longer term or just keep the best here.
4 participants
Routing wants to move traffic to the best model; provider prompt caches are per-model and only pay off when a conversation stays put. Every reroute silently throws away a warm cache and re-bills the whole prompt at the uncached rate — so a "cheaper" model can actually cost more.
This PR lets the two coexist: caching keeps conversations warm and cheap, and a per-session routing budget decides whether a proposed switch is worth the cache it gives up. The budget is a routing concern — it applies whether or not prompt caching is enabled — and it never overrides the router's quality decision.
Cache lookup
Independent of any cost decision, every turn Plano resolves which session a request belongs to and whether that session's provider cache is still warm:
X-Model-Affinityheader always wins. Otherwise Plano derives an implicit key by hashing (FNV-1a) the parts of the prompt that stay byte-for-byte identical across turns: the system prompt, the tool definitions, and the first user message. Conversation history grows at the tail, so those pieces don't change — turns 2+ hash to the same key. The first user message is in the key on purpose: without it, every session of the same agent (same system prompt + toolset) would collapse onto one pin and fight over a single model. The implicit key is derived whenever either prompt-caching affinity or the routing budget is active./routingdecision endpoint, where Plano never sees the provider's response.system + tools— the true cacheable prefix. If it changes mid-session, the provider cache is already gone: the session is treated as cold and re-binds.How it works
1. Automatic prompt caching (opt-in, zero-config)
OpenAI-family models cache a repeated prefix automatically. Anthropic-family models don't — they only cache if the request explicitly marks where the cacheable prefix ends, via a
cache_controlbreakpoint. So for Anthropic-family upstreams — including OpenAI-compatible gateways that front Anthropic, like DigitalOcean — Plano adds that breakpoint to the outgoing request itself. The upshot: a client that only speaks the OpenAI format still gets prompt caching on Anthropic models without having to knowcache_controlexists.2. Routing budget
The router still runs every turn. When it proposes a different model than the one a session is warm on, Plano calculates the actual input-token cost of the switch:
This is computed from input-token pricing only — output-token cost is deliberately excluded, since output length is unknowable before generation and betting on it is unreliable.
switch_cost ≤ 0) → free, always allowed. It never reduces the session's switch spend — the "saving" is vs a path we didn't take, not real money.max_overhead_pct% of the session's running never-switch baseline (what staying on the anchor would have cost). Once the cap is reached, Plano sticks. The promise: a conversation bills at mostmax_overhead_pct% above never-switching.The cap is yours to set; there are no baked-in defaults, and startup fails if it's configured without a
max_overhead_pct(or without a cost source).3. Observability for evals
Every decision is traceable. On the routing span:
plano.cache.warm— whether the session's provider cache was warm this turnplano.cache.idle_ms— idle gap (now − last_used) the warmth call was based onplano.switch.cost_in_usd— actual input-token cost of the proposed switch (negative when the candidate is outright cheaper)plano.switch.overhead_ceiling_in_usd— spend headroom for this switch (max_overhead_pct% × baseline)plano.switch.decision—allowedorretainedplano.switch.counterfactual_route— (optional) on a vetoed switch, the road not taken, for A/B analysisplano.session.overhead_pct— cumulative switching overhead consumed, as a % of the never-switch baseline (compare directly tomax_overhead_pct)plano.session.switch_spend_in_usd— cumulative $ spent on switches this sessionplano.session.baseline_in_usd— cumulative $ staying on the anchor would have cost (the denominator)plano.session.switches— switches taken so far this sessionplano.session.total_cost_in_usd— cumulative actual conversation cost (input + output), refined from real usage each turnPer-request cost also lands on each LLM span —
llm.usage.input_cost_usd,llm.usage.output_cost_usd,llm.usage.total_cost_usd— so a session's turns sum to its total. On the metrics side,brightstaff_session_switch_decisions_totalcarries areasonlabel (same_anchor/free/within_cap/over_cap/no_pricing).Config
The routing budget lives under
routingand is independent of prompt caching. Presence of the block turns it on.Opt out per request with
X-Plano-Cache: off. Full walkthrough + guide underdemos/llm_routing/routing_budget/.Notable design choices
routingand applies whether or notprompt_cachingis enabled.session_router::route()is shared by the proxy and decision endpoints, so they can't drift apart.Test plan
cargo test --lib(brightstaff + common + hermesllm) — pass (warmth truth table, budget depletion / free-credit / cold-reseed, handler parity, pricing regression)cargo clippy --all-targets --all-features -- -D warnings— cleanplano_config_schema.yamlX-Plano-Cache: offdisables per request